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Abstract 

This evaluation study assessed the effectiveness of an Emergency 
School Aid Act (ESAA) funded Secondary Reading Project (SRP) In aiding 
secondary students to overcome a lack of developmental skills in reading. 
Genera] procedures and specific techniques used In the evaluation are 
outlined and discussed. During the 1973-74 school year ESAA-funded SRP 
served students In twelve targeted high schools, while Dallas Independent 
School District (DISD) funded SRF served six nontargeced high schools. 
Included among the results were data indicating that (a) the most success- 
ful schools had numerous paraprof esslonals and volunteer tutors, thus 
reducing the student-adult ratio; (b) the lower the pretest achievement 
level, the higher the mean gain. 



us OEPARTMENTOFHEALTH, 
EDUCATION » WELFARE 
NATIONAL INSTITUTE OF 

EDUCATION „^„„^ 
TH.S OOCUWENT HAS BEEN REPRO 
OufFO EXACTLY AS RECEIVED FROW 
?^E PEPSON OR 0RGAN.2AT.cn OR.G.N 
ATr-G IT POINTSOF VIEW OR OPINIONS 
Cta^EO 00 NOT NECESS^«ILY RE^RE 
SENT OFFICIAL NATIONA. INSTITUTE OF 

EDUCATION POSITION OR POLICY 



O 

o 



Paper presented at the Annual Meeting of the American Educational 
Research Association, Washington, D. C, April, 1975. 




EVALUATION OF AN ESAA 
SECONDARY READING PROJECT 



Robert P. Grobe & Melinda A. Rnecht 
Dallas Independent School District 

An evaluation study was conducted to assess the effectiveness of an 
Emergency School Aid Act (ESAA) funded Secondary Reading Project (SRP) in 
aiding secondary students to overcome a lack of developmental skills in 
reading* This paper summarizes information obtained in that evaluation* 
More specifically, the objectives of this paper are (a) to present an over- 
view for SRP as it was implemented and conducted in the Dallas Independent 
School District (DISD), (b) to discuss the evaluation model used to evalu- 
ate SRP, and (c) to present major results revealed in the evaluation of 
SRP. 

Project Overview 

The DISD is committed to provide special reading services to all students 
of all ethnic backgrounds to help them achieve their reading potential. 
Prior to 1973, many students, particularly minority students, fell far be- 
low the national norms in reading achievement. The Secondary Reading Pro- 
ject CSRP) was founded on the belief that every secondary student should 
be given every opportunity to develop to his potential. Thus, the project 
was organized to provide for the planning, developing, implementing, and 
evaluating of a continuous program of reading instruction for certain 
secondary students who perform below the level of functional literacy. 
Stated goals included an improved self-concept and a one month increase in 
reading achievement for each month of instruction. 

During the 1973-74 school year, SRP served students in twelve targeted 
high schools, six nontargeted high schools, and twenty-three junior high 



schools. Table 1 presents a list of the SRP schools with the number of 
full-time reading teachers per building. In addition to these full-time 
reading teachers, students in the twelve targeted high schools had the 
teamwork services of a teacher aide, a tutor, and a school volunteer in 
each classroom. Each senior high school received $1,000 per reading 
teacher for the purchase of supplies and materials, while junior high 
schools received $500 per reading teacher. 

SRP had four components: Special Remedial Services , Curriculum 
Development , Administration , and Evaluation . Special Remedial Services 
included three subcomponents: Instructional, Staff Training, and 
Community Affairs. The primary goal of the Instructional subcompoiient was 
to aid secondary students of all backgrounds in overcoming a lack of 
developmental skills in reading. Individualization of learning and 
teaching was a primary concern, and all reading materials were adopted 
for personalized learning. 

The Staff Training subcomponent provided special training for sec- 
ondary principals, leadership staff, and classrooio reading teachers to 
implement reading instruction appropriate to the individual needs and 
abilities of studerkts from diverse socioeconomic backgrounds. The overall 
objective of this subcomponent was to develop a cadre of personnel highly 
trained in implementing multilevel, multiethnic, and multisensory reading 
programs. 

Tne Community Affairs Subcomponent was designed to complement the 
objective in reading. Significant efforts were made to inform and involve 
the home and community in the school program in order to help students 
achieve their maximum potential. Specifically, two home-school liaisons 



Table 1 



SRP Schools and Number of Full-Time Reading Teachers per Building 





Targeted High Schools 




Bryan Adams 


5 


Franklin D. Roosevelt 


7 


W. H. Adamson 


3 


W. W. Samuell 


3 


Hillcrest 


3 


South Oak Glif f 


7 


Lincoln 


7 


H. Grady Spruce 


5 


North Dallas 


7 


. W. T. White 


3 


L. G« Pinkston 


7 


Woodrov Wilson 


3 


Nontargeted High Schools 




David W. Carter 


2 


Seagoville 


1 


Thomas Jefferson 


2 


Skyline 


2 


Justin F. Kimball 


2 


Sunset 


2 




Junior High Schools 




W. H. Atwell 




D. A. Hulcy Middle 


1 


T. W. Browne 




J« L. Long 


1 


E. H. Gary 




Thomas G. Marsh 


1 


E. B. Comstock 




T. J. Rusk 


1 


Thomas A* Edison 




Rylie Middle 


1 


Fred F. Florence 




Sequoyah Middle 


1 


Benjamin Franklin 




Alex W. Spence 


1 


W. H. Gaston 




L. V. Stockard 




W. E. Greiner 




Boude Storey 




Robert T. Hill 




E. D. Walker Middle 




0. W. Holmes 




Sarah Zumwait 




John B. Hood 
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were assigned to each targeted high school • They worked in the communlcy 
attempting to get all students to school and then to keep them there. 

The major functions of the Curriculum Development component were ^o 
write and implement a sequential 7-12 curriculum which would provide a 
balanced course of study by incorporating a range of peoples and cultures. 

The Adminis t rat ion component provided office and support staff to 
assist attainment of project objectives in a cost-effective manner. The 
component provided technical assistance to project personnel in the plan- 
ning, implementation, and general management of the project. 

The objectives of the Evaluation component were to delineate, obtain, 
and provide useful descriptive and formative feedback for process decision- 
making. Additionally, this component provided end-of-year sumraative infor- 
mation for determining the degree to which program operation conformed to 
program design and the extent to which components met their stated objec- 
tives. 

Theoretical Framework 

Evaluation of SRP was rather loosely based upon Stuf flebeam's Context- 
Input-Process-Product (CIPP) evaluation model. All types of evaluation 
were addressed in the SRP evaluation; however, process evaluation was 
emphasized during the first year of implementation. General steps in the 
evaluation of SRP are outlined below: 

1» The evaluator worked with project management to formulate 
evaluation questions. 

2* The project director examined and reacted to the evaluation plan. 

3. Students were pretested on affective and cognitive variables, 

4* Process evaluation activities were conducted and culminated in 
twenty Interim reports. This step also Involved much interaction 
between the evaluator and the project staff concerning findings 
from the process evaluation, 
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S* Context and input evaluation activities were conducted as needed 
and/or as time allowed* 

6. Students were posttested on affective and cognitive variables* 

7* Product evaluation Information for determining the degree to which 
program operation conformed to program design and the extent to 
which components met their stated objectives was summarized and 
presented in a final report* 

8* The final report was used by the project manager as a basis to 
formulate recommendations for the next cycle of the project* 
These recommendations were presented to the local school board* 

These steps outline the basic framework of this evaluation; however » 
they do not convey the dynamic scramble In arriving at the final report 
for the school board* A brief description of this evaluation should 
Illustrate what is probably a general pattern for evaluations of many first 
year» federally funded projects* 

The evaluator arrived on site during June* Planning for project Imple-* 
mentation had been completed and a month-long summer workshop was underway. 
A tentative evaluation plan was required immediately* Since a pretest-post- 
test design was chosen » an Instrument needed to be selected and ordered* 
Concurrently^ descriptive and definitive information had to be acquired in 
efforts to conceptualize the project as an integrated system. Also during 
the same first months » considerable effort was devoted to monitor ing-process- 
formative activities to enhance the probability of a quick startup and 
eliminate the possibility of evaluating a nonexistent project* Since more 
time was available after the pretest administration » the evaluation plan was 
then finalized » specified evaluation activities were Implemented, and 
planning for the next year's evaluation was initiated* 
Methods and/or Techniques 

In effecting this evaluation » the following methods and techniques were 

used: 



1. Interaction and Service*- the Impact of this evaluation was affected 
by the degree of credibility that the evaluator had with the project 
staff. It was found that the evaluator* 8 credibility was enhanced 
by frequent Interaction with project staff. As Interim or process 
evaluation reports were written, they were suomarlzed at project 
staff meetings. Additionally, the evaluatoJ*s credibility was In- 
creased to the degree that a service could be provided. To 
provide a service the evaluation design must have enough flexibility 
to occasionally obtain idiosyncratic information. 

2. Observation — considerable effort was expended upon classroom 
observation during the initial implementation phase. This monitoring 
was deemed necessary so that startup delays might be minimized. 
Additional classroom observations were made throughout the year to 
determine the kinds of materials and the degree of individualization 
used in SRP classrooms. 

3. Surveys — several paper and pencil surveys were used to determine in- 
formation which could be used as guides to areas where more depth 
information might be useful. As an example, content area teachers 
were surveyed to determine their awareness of SRP and the services 
available from SRP. 

4. Interviews — personal Interviews were conducted in cases where a less 
structured form of response was likely to reveal valuable information 
concerning bottlenecks in paperwork, personnel problems, or construc- 
tive suggestions. 

5. Testing — pretests and posttests were administered in the effective 
and cognitive domains. Pretest results were used in a process 
evaluation report to pinpoint inequities in the assignment of staff 
to particular buildings. Fosttest results were utilized in the final 
product evaluacioit. 

6. Systems Analysis — I .mlted systems analysis techniques were used in 
attempts to determine if bottlenecks existed, if replication of 
effort occurred, and if project components interfaced effectively. 

7. Statistical Analysis — statistical analyses were used with affective 
and cognitive data in the product evaluation. 

Data Source 

This evaluation used a small sample (approximately 10%) of SRP students 
with Comprehensive Teots of Basic Skills (CTBS) total reading scores. The 
stipulation placed on this stratified random sample was that students had to 
have comp'ete data which consisted of sixth*-, seventh-, and eighth-grade 
scores. All students with pretest and posttest scores on the Gates-MacGlnitie 
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Reading Tests (GMRT) were included for analysis. Affective measures were 
administered to each teacher* s first period class. In addition, pretest- 
posttest affective measures were taken for one class of each of 10 randomly 
selected teachers* 

All SRP teachers and resource teachers were asked to respond to several 
surveys, and were also observed with the aid of a behavioral checklist. A 
nonstructurad interview was conducted with a sample of principals. A ten 
percent stratif ied~random sample of parents was surveyed via U.S. Mail. In 
addition, administrators, content area teachers, and community members were 
sur»7eyed or interviewed concerning their awareness ci. the ESAA reading pro- 
ject. 

Evaluation Desig n 

The evaluation design used for the CTBS uotal reading scores consisted of 
the following repeated measures paradigm: 

Ol Xt O2 Xt O3 XgRp O4 

The observations (Oj^, O3), consisted of the CTBS total reading scores; 
from a student* s sixth-, seventh-, and eighth-grade years; 0^ was the CTBS 
total reading score achieved during the 1973-74 school year; Xij, was the 
traditional English class, and Xgj^ was the SRP reading class. 

The design used for evaluation of the GMRT data consisted of a two- 
group, pretest-post test design which is schematically represented as: 

Ol O2 
Ol X2 O2 

The observations (0^, O2) consisted of pretest and posttest data, respectively; 
X^ represents the ESAA-SRP treatment in targeted high schools, and X2 represents 
the DISIHSRP treatment in nontargeced high schools* The regression analysis 



approach was employed for the GMRT component of this analysis. Within the 
regression analysis framework, the GHRT posttest served as the criterion 
variable, with the GMRT pretest serving as a covarlate. Predictor variables 
such as project type, student sex, and student ethnicity were used to cate- 
gorize the data to test hypotheses on their effectiveness in accounting for 
pupil achievement. 

The design used for evaluation of affective data Included the following 
two-group posttest paradigm: 

Xi 02 
X2 O2 

The observations (O2) consisted of posttest measures; Xj^ and X2 represent 
different subgroups of the population, such as targeted-nontargeted or 
male-female. 

In addition, a pretest-posttest paradigm similar to that used with the 
cognitive data was used to obtain affective data for a small sample of SRP 
students* 

The major evaluation questions of interest are delineated by component 
below: 

A. Secondary Reading Department 

1. Did achievement results vary among the SRP schools? 

2. Did SRP students at different achievement levels on the pretest 
exhibit differential gains in reading achievement at the end of 
the year? 

3. Did students in SRP targeted high schools performs on a selected 
standardized measure of reading achievement at a level equivalent 
to a sample of students in nontargeted high schools? 

4. How did student reading achievement gains under SRP compare to 
past student reading achievement gains without SRP in targeted 
high schools? 
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5. Was reading achievement affected by student-adult ratio? 



6. How did targeted SRP, nontargeted SRP, and nonSRP English students 
compare in terms of attitude-toward-self , attitude-toward-school, 
and attitude-toward' reading? 

7. Hew did students perceive the assistance they received from SRP 
classes? 

B. Staff Training 

8. What was the perceived effect of pteservice and inservice staff 
development workshops? 

C. Community Affairs 

9. What was the role of the home-school liaison? 

10^ Were stated objectives for the home-school liaisons achieved? 

11. How many volunteer parents or volunteer tutors worked in SRP 
classes? 

D. Curriculum Development 

12. What efforts were made in the area of Curriculum Development? 

13. How did teachers react to the Multiethnic Studies Program (MSP) 
material? 

E . Adminl s t r a t ion 

14. Was assistance supplied to other project components when requested? 

15. Were ESAA Advisory Connnittee members assisted in performing their 
role? 

Evaluation Results ; 

A. Secondary Reading Department 

1. SRP was actually 12 different projects, although in many cases 
only slight modifications were evidenced. During the eight 
month pretest-posttest interval, different results were achieved 
at each site; the more successful projects attained mean grade 
score gains in vocabulary achievement of 1.34, 1.19, 0.79, and 
0.63. The common thread for these schools seems to be relatively 
low student-adult ratio. 

2. Overall, students at different pretest achievement levels 
exhibited differential gains in achievement for both vocabulary 
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and comprehenelon. As revealed In Table 2, an inverse relation- 
ship seems to exist; that the lower the pretest achievement level, 
the higher the mean gain for that group* 



Table 2 



GMRT 


Mean 




Mean 




Pretest 


Vocabulary 




Comprehension 




Achievement 


Gain in 




Gain In 




Level 


Grade Score* 


N 


Grade Score* 


N 


1.1-1.9 


1.10 


164 


1.25 


173 


2.0-2.9 


0.62 


354 


0.84 


330 


3.0-3.9 


0.68 


496 


0.69 


501 


4.0-4.9 


0.62 


560 


0.45 


576 


5.0-5.9 


0.37 


797 


0.31 


915 



*^Gains were achieved during an eight month interval • 



3. Students in the targeted schools started (pretest) and ended 
(posttest) at a lower reading achievement level, tL^n the non- 
targeted students; however, the targeted students showed a larger 
gain in reading achievement. No statistical differences were 
noted between means for students in targeted and no-» argeted 
schools on either the GMRT vocabulary or comprehension subtests, 
when the entry achievement level (pretest score) was used as a 
covarlate. A summary of the results is displayed in Table 3: 



Table 3 









GMRT 


GMRT 








Pretest 


Posttest 


Subtest 






Mean 


Mean 


Vocabulary 


Targeted 
Nontargeted 


1922 
421 


34.2 
37.5 


37.8 
40.4 


Comprehension 


Targeted 
Nontargeted 


2155 
496 


31.5 
34.6 


34.9 
36.9 



4. A stated objective for SRP was worded as follows: "Students will 
gain one month in achievement for each month in the program." 
This objective was not attained in SRP; ar one-third of the 
students exhibited a gain of 8 months or more in achievement 
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during 8 months^ instruction. Another way of examining the 
data in termp of months of instruction was to compare the 
grand mean pretest and posttest scores. Such a comparison 
reveals a mean Increase of 7 months' achievement in vocabulary 
for 8 months' instruction and 5 months' achievement in com- 
prehension for 8 months' instruction. 

5. Historical or longitudinal CTBS data for a siu^x^ sample (10%) of 
students revealed a linear trend in achievement across time. That 
is, the information indicated that, overall, students gained at 
the same rate in past years as they did during the 1973-74 school 
year. There were, of course, several programs that demonstrated 
dramatic gains. 

6. There seemed to be some indication that small student-adult ratios 
had a positive, enhancing affect upon achievement gains; however, 
the relationship did not hold without exception across all schools. 
Large gains were evident in the schools with numerous volxmteers, 
where the students-adult ratios vere greatly reduced. 

7. Attitude-toward-self , attitude-toward-school, and attitude-toward- 
reading were investigated with a battery of affective surveys. 
Significant differences were found among mean scores for targeted 
SRP students, nontargeted SRP students, and nonSRP English students. 
NonSRP English students attained the highest mean score In each 
case; however, the differences were not large enough to reflect 

any great practical significance. In addition, pre-post analyses 
revealed no significant effect of the project on attitudes. 

8. During classroom observations, most students appeared satisfied 
Tyith the assigned work. Many of the students apparently felt that 
the materials and assistance were of value. 

Staff Training 

9. Teachers seemed to feel that the summer workshops vere useful. 
However, inferences from other information presented in this 
report plus anecdotal information from conversations indicated 
that the in-service programs presented during the year were, in 
general, not well received. For future workshops, reading 
teachers requested allocation of time for "hands-on" experience 
with new materials and equipment. They also wanted organized 
sessions in which knowledgeable persons illustrate the advantages 
and disadvantages of particular programs, equipment, and techniques. 

Community Affairs 

10. The role of the home-school liaison included helping to interpret 
the reading program to parents and the community, establishing 
appropriate two-way communications between home and school, and 
working with students to encourage regular school attendance. 




11. Home-school liaisons met predetermined evaluative criteria for two 
of eight stated objectives. However, they were close to meeting the 
evaluative criteria on two of the remaining objectives. Two of the 
objectives with related evaluative data are displayed in Table 4. 







Table 4 








Oblective 


Evaluative 
Data 


Objective 
Attained 






The home-school liaison 
attends games, PTA 
programs, and other 
school functions. 


90% of resource 
teachers indicated 
that such activities 
were attended. 


Yes 






The home-school liaison 
team visits the homes 
of all students in SRP 
at least once. 


19% of the parents 
suirveyed indicated 
they were visited. 


No 





12. The distribution of volunteer parents among the targeted high 
schools was extremely disproportionate. There was a range of 
0 to 81: two schools had a large number of volunteers (54 and 
81); five schools had between 2 and 9; five schools did not 
report having any volunteers. 

D. Curriculum Component 

13. Curriculum writers are in the process of writing and implementing 
a sequential 7-12 curriculum for the 1974-75 school year with the 
stated goal of providing a balanced social science curriculum 
which incorporates a range of peoples and cultures. 

14. Responses among Multiethnic Studies Program (MSP) teachers 
indicated positive attitudes toward the program. 

E. Administration 

15. The Administration Component was judged by all project managers 
to supply needed assistance when requested. 

16. All committee members indicated that ESAA project managers were 
assisting them to lulfill their role. 

Educational Importance of the Study 

Informatiori was obtainetl concerning the organization of and results 



attributed to a large secondary reading project* Similar secondary reading 
projects may be Interested In using this Information to build upon, thus 
enabling them to bypass rlme-sappli*g, organizational and Instructional pro- 
blems* Thle evaluation might be used as a starting point £or future evalu- 
ations of secondary reading projects. In addition, evaluators can use the 
data presented for general comparisons* 
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